Exact and Perturbation Solutions for the Ensemble Dynamics

This paper presents two approaches to characterize the dynamics of weight space probability density starting from a master equation In the rst we provide a class of algorithms for which an exact evaluation of the integrals in the master equation is possible This enables the time evolution of the density to be calculated at each time step without approximation In the second we expand earlier work on the small noise expansion to a complete perturbation framework As an example application we give a perturbative solution for the equilibrium density for the LMS algorithm Finally we use the perturbation framework to review annealed learning with schedules of the form t t p